Information Elicitation Sans Verification

The recent advent of human computation — employing groups of non-experts to solve problems — has motivated study of a question in mechanism design: How do we elicit useful information when we are unable to verify reports? Existing methods, such as peer prediction and Bayesian truth serum, require assumptions either on the mechanism’s knowledge about the participants or on the information structure of participants for eliciting private information from the participants. Meanwhile, however, there are simple mechanisms in practice such as the ESP Game that seem to require no such assumptions, yet have achieved great empirical success. We attack this paradox from two directions. First, we provide a broad formalization of the problem of information elicitation without verification and show that, without assumptions on designer knowledge or participants’ information, there do not exist mechanisms that can truthfully elicit the private information of the participants for this setting. Second, we define and analyze the output agreement class of mechanisms, an extremely broad but simple mechanism in which players are rewarded based on the metric distance between their reports. Output agreement makes no assumptions on designer knowledge or participants’ information and thus cannot be “truthful”. We resolve the paradox by showing that it is useful: It elicits the correct answer according to the common knowledge among the players. We conclude with an analysis of the assumptions and results of various popular mechanisms for information elicitation without verification.